Added `delete_tileset` manage command and `tileset` API route #79

alexpreynolds · 2018-11-13T07:56:37Z

Here is a demonstration tileset:

$ python manage.py list_tilesets
tileset: Tileset [name: LN43287.75_20.normalized.GRCh38_no_alts.bw.tileset] [ft: hitile] [uuid: AIVpsJYwSemD8FVPBv6vrw]
$ ls -al higlass-server/media/uploads/
total 207764
drwxrwxr-x 2 ubuntu ubuntu      4096 Nov 13 06:15 ./
drwxrwxr-x 3 ubuntu ubuntu      4096 Nov 13 06:15 ../
-rw-rw-r-- 1 ubuntu ubuntu 212742028 Nov 13 06:15 LN43287.75_20.normalized.GRCh38_no_alts.bw.tileset

Here is an example showing the use of the manage.py delete_tileset command to remove this tileset file and database record:

$ python manage.py delete_tileset --uuid=AIVpsJYwSemD8FVPBv6vrw
$ python manage.py list_tilesets
$ ll higlass-server/media/uploads/
total 8
drwxrwxr-x 2 ubuntu ubuntu 4096 Nov 13 06:52 ./
drwxrwxr-x 3 ubuntu ubuntu 4096 Nov 13 06:15 ../

The API call uses the DELETE method and the IsAuthenticated permission class decorator, which requires authentication. Here is an example of an (unauthenticated) API call:

$ python manage.py list_tilesets
tileset: Tileset [name: LN43287.75_20.normalized.GRCh38_no_alts.bw.tileset] [ft: hitile] [uuid: MSLMji4TSVKynUGNui-5cQ]
...
$ curl -X "DELETE" http://localhost:8000/api/v1/tileset/?d=MSLMji4TSVKynUGNui-5cQ
{"uuid": "MSLMji4TSVKynUGNui-5cQ"}

The API returns 40x errors if the underlying instance or file is not found or cannot be removed.

For debugging purposes, I modified the Tileset model __str__ function to return the tileset's uuid along with its name and filetype properties.

…ndles DELETE request

…oute

alexpreynolds · 2018-11-13T11:06:51Z

I added a management command and API route to modify a tileset's properties by uuid. Currently, this only modifies the name field (if specified), but this could be expanded to modify other tileset properties with additional logic.

Here is an example management command:

$ python manage.py modify_tileset --uuid="Zfg_ZtU0TZeJYRoRQwRMFQ" --name="bar"

And here is an example API call:

$ curl -H "Accept: application/json" -X "POST" -d '{"uuid":"BAeHd65pSCmI8Cs-xS5ACw", "name":"foo"}' http://localhost:8000/api/v1/tileset/

pkerpedjiev · 2018-11-13T14:38:06Z

This is great! Thanks Alex.

A few things before merging:

As mentioned in the inline comments, could you check if the DELETE endpoint doesn't already exist in TilesetsViewSet?
The documentation for higlass_server is here: https://github.com/higlass/higlass/blob/develop/docs/higlass_server.rst. Would you mind adding a few lines describing the new API endpoints?
It looks like you've tested this quite extensively. To make sure we don't introduce regressions, could you just copy those manual tests to our automated test section here: https://github.com/higlass/higlass-server/blob/master/higlass_server/tests.py?

Thanks!

pkerpedjiev · 2018-11-13T14:32:31Z

tilesets/views.py

@@ -492,6 +494,95 @@ def tiles(request):
        )

    return JsonResponse(tiles_to_return, safe=False)
+
+
+@api_view(['POST', 'DELETE'])


Did you check if the curl -X 'DELETE' server:port/api/v1/tilesets/[uuid]/ endpoint already exists? There's already a tilesets viewset that should provide, GET, POST, UPDATE, DELETE endpoints here:

higlass-server/tilesets/views.py

Line 683 in dc4d380

class TilesetsViewSet(viewsets.ModelViewSet):

I know the POST endpoint works because I've been using it to upload datasets to higlass.io. Take a look here: http://docs-dev.higlass.io/data_preparation.html#importing-into-higlass

If possible, I'd prefer to avoid duplicating API endpoints.

pkerpedjiev · 2018-11-13T14:39:48Z

README.md

@@ -55,6 +55,34 @@ curl http://localhost:8000/api/v1/tileset_info/?d=hitile-demo
 curl http://localhost:8000/api/v1/tiles/?d=hitile-demo.0.0.0
 ```

+## Preparing files for ingest


All of the datatype specific documentation is in the main documentation repository: http://docs-dev.higlass.io/data_preparation.html

The code for that can be found here: https://github.com/higlass/higlass/tree/develop/docs

pkerpedjiev · 2018-11-13T14:42:05Z

requirements.txt

@@ -1,8 +1,8 @@
-Cython==0.25.2
+-e git+https://github.com/mirnylab/cooler.git@master#egg=cooler


What's in the master branch that isn't in v0.7.10?

alexpreynolds · 2018-11-15T17:04:35Z

Sorry. I'm going to close this PR and start over.

pkerpedjiev · 2018-11-15T18:10:19Z

No, it's my fault that we don't have better documentation about the API endpoints. Ideally the docs would have a section on the REST API. Without looking at the code it's nearly impossible to figure out what endpoints exist and how to access them.

All this is to say, your PR is really appreciated and hopefully it's not too much work to fit it into TilesetViewSet.

alexpreynolds · 2018-11-15T18:18:52Z

I neglected to sync my fork with upstream, so starting over was necessary, anyway.

Looks like the built-in DELETE does remove the database entry, but it does not remove the tileset file from the media/uploads directory. I'll look at overriding the behavior of the destroy() method to take care of that, so that it would behave the same as the management command.

Ubuntu and others added 3 commits November 13, 2018 07:42

Added delete_tileset manage command and tileset API route that ha…

e0bce30

…ndles DELETE request

Added modify_tileset management command and POST tileset modify r…

7dc5683

…oute

Reenabled authentication flag

04dd014

alexpreynolds and others added 3 commits November 13, 2018 03:36

Merge branch 'develop' into delete-tileset

08a86d9

updated requirements manifests

a00ff43

pybbi should point to the released version

e622c42

pkerpedjiev requested changes Nov 13, 2018

View reviewed changes

alexpreynolds closed this Nov 15, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added `delete_tileset` manage command and `tileset` API route #79

Added `delete_tileset` manage command and `tileset` API route #79

alexpreynolds commented Nov 13, 2018 •

edited

alexpreynolds commented Nov 13, 2018 •

edited

pkerpedjiev commented Nov 13, 2018

pkerpedjiev Nov 13, 2018

pkerpedjiev Nov 13, 2018

pkerpedjiev Nov 13, 2018

alexpreynolds commented Nov 15, 2018

pkerpedjiev commented Nov 15, 2018

alexpreynolds commented Nov 15, 2018

		@@ -1,8 +1,8 @@
		Cython==0.25.2
		-e git+https://github.com/mirnylab/cooler.git@master#egg=cooler

Added delete_tileset manage command and tileset API route #79

Added delete_tileset manage command and tileset API route #79

Conversation

alexpreynolds commented Nov 13, 2018 • edited

alexpreynolds commented Nov 13, 2018 • edited

pkerpedjiev commented Nov 13, 2018

pkerpedjiev Nov 13, 2018

Choose a reason for hiding this comment

pkerpedjiev Nov 13, 2018

Choose a reason for hiding this comment

pkerpedjiev Nov 13, 2018

Choose a reason for hiding this comment

alexpreynolds commented Nov 15, 2018

pkerpedjiev commented Nov 15, 2018

alexpreynolds commented Nov 15, 2018

Added `delete_tileset` manage command and `tileset` API route #79

Added `delete_tileset` manage command and `tileset` API route #79

alexpreynolds commented Nov 13, 2018 •

edited

alexpreynolds commented Nov 13, 2018 •

edited